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NEPTUNE:  Helping  Program 
Managers  Understand  their 
Program  Customers 


Zannette  A.  Uriel! 

Institute  for  Organizational  Assessment  (PERS-14) 

Michael  J.  Schwerin,  Ph.D. 

Research  Triangle  Institute  International 


The  Chief  of  Naval  Operations  (CNO)  has  said,  “I  intend  to  lead  a  Navy  that  holds 
quality  of  service  for  Sailors,  for  their  quality  of  life  and  their  quality  of  work,  as  a  top 
priority  in  mission  and  combat  readiness.”  (CNO,  2000)  Additionally,  studies  have  found 
that  meeting  non-work  quality  of  life  (QOL)  needs  impacts  retention  intentions 
(Koopman  &  Goldhaber,  1997;  Wilcove,  Wolosin,  &  Schwerin,  2002).  This  paper 
presents  a  prototype  decision  support  system  (DSS),  entitled  the  Navy  Evaluation 
Program  to  Track  User  Needs  Electronically  (NEPTUNE),  that  was  created  for  a  study 
to:  1)  determine  a  program  evaluation  methodology  encompassing  all  Navy  QOL 
programs  and  2)  find  program  impact  on  the  military  outcomes  of  readiness  and 
retention. 
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Background  -  Quality  of  Life  Studies 

•  Commanders  always  have  been  interested  in  well-being  of 
personnel 

•  In  early  1990s,  first  formal  studies  of  Navy  Quality  of  Life 
(QOL)  conducted 

-  Needs  based 

•  Later  studies  conducted  to  determine  a  global  QOL  score 

•  In  late  1990s,  studies  tried  to  link  programs  to  QOL 
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Good  commanding  officers  have  always  been  interested  in  the  well-being  of  their 
Sailors.  They  may  have  asked  their  users  for  feedback,  either  formally  (e.g.,  comment 
cards,  customer  surveys)  or  informally  (e.g.,  word  of  mouth).  Although  helpful,  this  does 
not  lend  itself  to  easily  determining  where  to  best  allocate  resources  to  achieve  the 
greatest  effect  on  Sailors  and  their  families. 

With  the  All-Volunteer  Force,  the  Navy  has  become  more  sensitive  to  QOL  in  order 
to  compete  with  the  civilian  sector.  In  the  1990s,  some  of  the  headquarters  QOL  divisions 
began  conducting  Navy-wide  assessments;  Morale,  Welfare  and  Recreation  (MWR)  and 
Fleet  and  Family  Service  Centers  (FFSC)  both  conducted  surveys  to  determine  program 
use  and  satisfaction.  Additionally,  the  Navy  began  studying  the  topic  of  QOL  itself,  with 
the  first  studies  attempting  to  identify  needs  of  Sailors  and  Marines  (Booth-Kewley  & 
Thomas,  1993),  and  later  studies  trying  to  find  a  relationship  between  needs  and  the 
concept  of  global  QOL  (Wilcove,  1996)  and  then  QOL  and  career  intentions  (Kerce, 

1995;  White,  Baker,  &  Wolosin,  1999;  Wilcove  &  Schwerin,  2002). 

In  the  late  1990s,  Kerce  (1998)  and  Kerce,  Sheposh,  and  Knapp  (1999)  attempted  to 
find  empirical  links  between  programs.  To  do  this,  Kerce  (1998)  created  a  list  of  Reasons 
for  Being — statements  that  might  indicate  a  program’s  purpose,  such  as  providing 
support  to  allow  members  to  concentrate  on  their  mission.  Surveys  with  questions  based 
upon  these  Reasons  for  Being  were  administered  at  select  sites.  Kerce  et  al.  (1999)  found 
that  this  methodology  was  effective  overall,  despite  problems  with  a  lack  of  objective 
data  and  a  limited  number  of  respondents. 
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Background  -  DSS  in  the  Navy 

•  Historically  DSS  primarily  used  at  the  headquarters  level 

-  Used  often  for  planning  and  forecasting  Navy  manpower 

•  Generally  has  not  been  used  for  QOL  programs 

-  Most  headquarters  programs  conduct  data  calls  as 
needed 

-  QOLMIS 
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Historically  DSS  have  been  used  at  the  headquarters  level,  usually  for  planning  and 
forecasting  manpower  and  for  personnel  needs  and  requirements.  With  increases  in 
information  technology  (IT)  capabilities,  they  have  become  much  more  useful  and 
common  in  the  military.  The  Air  Force  has  a  system  for  its  equal  opportunity 
assessments;  data  from  individual  bases  or  commands  can  be  “rolled  up”  to  higher  levels 
for  analysis.  The  Navy’s  Argus  system  gathers  exit/retention  survey  data  that  can  be  used 
at  a  variety  of  levels. 

QOL  was  not  one  of  the  early  adopters  of  DSS.  Traditionally,  if  data  were  required  to 
answer  an  internal  Navy  tasker  or  a  Congressional  inquiry,  a  data  call  was  sent  out  and 
bases  would  send  their  data  back  to  headquarters.  There  was  a  system  created  for  a  small 
number  of  QOL  programs  called  Quality  of  Life  Management  Information  System 
(QOLMIS),  but  this  system  was  not  intuitive  and  some  felt  it  was  not  useful. 
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Allow  managers  at  diverse  locations  and  at  different 
hierarchical  levels  to  determine: 

•  How  Sailors  rate  programs  on  quality 

•  How  well  each  program  meets  primary  objectives 

•  Which  programs  have  greatest  impact  on  outcomes 

•  Differences  between  locations 
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The  primary  objectives  of  the  DSS  included  determining:  1)  how  Sailors  rate  the 
different  programs  on  their  quality  (i.e.,  hours,  facilities,  range  of  services/programs, 
customer  service,  quality  of  service,  and  value  of  services  in  relation  to  cost);  2)  how  well 
each  program  meets  its  primary  objectives  (see  Appendix  A  for  a  list  of  Reasons  for 
Being  used  in  this  study);  3)  which  programs  have  the  greatest  impact  on  outcomes  (i.e., 
QOL,  personal  readiness,  and  career  intentions);  and  4)  what  differences  there  are 
between  Navy  locations. 
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Challenges 


NPRST 


•  Variety  of  Navy  Quality  of  Life  (QOL)  programs 


Bachelor  Housing 
Child  Development 
CREDO 

Deployment  Support 
Family  Housing 
Food  and  Hospitality 


Navy  College  Program 
Personal  Financial  Management 
Recreation  and  Fitness 
Relocation  Assistance  Program 
Spouse  Employment  Assistance  Program 
Youth  Programs 


•  Wide  variety  of  sponsors 

•  Lack  of  objective  data 

•  Lack  of  “roll-up”  for  some  programs 
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QOL  within  the  Navy  is  complex.  Because  of  the  unique  characteristics  of  Navy  life, 
Sailors  and  their  families  are  always  being  impacted  by  non-work  Navy  programs  (see 
Appendix  B  for  a  listing  of  programs  used  in  this  study). 

Because  of  the  military  hierarchy,  there  are  a  number  of  levels  of  oversight  for  each 
of  these  programs.  Each  program  has  its  own  manager  at  the  headquarters  level.  There 
are  overall  QOL  managers  at  the  claimant  level  (determined  by  mission)  and  at  the 
regional  level  (determined  by  geography).  The  actual  day-to-day  operations  of  the 
programs  are  determined  at  each  base,  with  program  managers  reporting  to  the  base 
commanding  officer.  Each  of  these  levels  requires  different  amounts  of  detailed 
information. 

Some  of  the  recreational  programs  primarily  raise  money  for  other  programs,  and 
have  therefore  primarily  tracked  monetary  data.  Others  are  certified  on  a  recurring  basis 
and  so  are  required  to  have  data  on  a  number  of  factors.  There  are  not  consistent 
objective  measures  across  all  programs. 

Because  the  funding  and  managing  of  programs  has  been  delegated  to  lower  levels, 
sometimes  higher-level  managers  do  not  have  knowledge  about  customer  opinions  at  the 
base  level.  When  required  to  justify  a  program,  the  program  managers  might  conduct  data 
calls,  where  they  request  information  from  individual  bases  about  select  topics.  However, 
these  data  calls  generally  are  on  an  as-needed  basis,  to  answer  a  specific  question. 
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Decision  Support  System;  NEPTUNE 

•  Original  plan  was  to  compare  programs  and  locations  based 
upon  key  questions,  cost  per  user 

-  Some  key  data  not  obtainable 

•  Modified  system  allows  users  to  query  to  fit  their  needs 

•  Single  database  with  multiple  access  levels 

-  Different  amounts  of  information  available  at  each  of  the 
levels 

Na\y  Personnel  Research,  Studies,  &  Technology  < 


The  original  concept  for  a  decision  support. system  (DSS)  was  to  allow  users  to  input 
a  wide  variety  of  data  and  then  generate  an  index  score  for  each  of  the  programs  to  allow 
for  comparisons.  When  index  scores  were  not  possible  because  of  the  lack  of 
quantitative,  objective  data  for  many  of  the  programs,  the  DSS  was  only  populated  with 
the  survey  data.  This  was  done  because  the  survey  data  was  more  information  than  some 
had  access  to  previously  and  the  survey  data  met  recent  policy  requirements  for  funding; 
users  could  query  the  system  to  find  the  survey  information  they  needed. 

After  introductory  screens,  NEPTUNE  asks  for  a  user  name  and  password.  The 
system  is  set  up  to  allow  users  to  only  access  information  that  directly  relates  to  them. 
For  example,  a  base  user  in  Norfolk  is  allowed  access  to  that  base’s  information  as  well 
as  a  composite  for  all  the  bases  within  that  region.  There  are  four  access  levels  of  the 
database  that  are  based  on  geographical  location,  from  base-level  to  entire  Navy;  at  each 
of  these  access  levels  users  may  see  more  data.  There  is  also  a  level  that  is  not  based  on 
geography;  program  rhanagers  at  the  headquarters  level  are  able  to  see  all  data  for  their 
particular  program. 

Since  this  is  a  prototype,  the  following  screenshots  reflect  only  those  few  locations 
where  surveys  were  administered.  Therefore,  if  all  PACFLT  is  selected,  the  information 
displayed  in  the  screenshot  only  reflects  those  bases  where  data  were  collected. 
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Once  within  the  system,  users  can  select  a  program  and  geographical  location.  They 
can  also  filter  the  system  with  respondent  status  and  up  to  three  other  questions;  for 
example  in  Recreation,  users  can  select  only  active  duty  respondents  who  used  the  on- 
base  gymnasium  once  a  week  and  a  civilian  facility  once  a  week.  Once  any  filters  are 
selected,  a  desired  analysis  question  must  be  selected.  These  include  the  quality 
questions,  program  impact  questions,  and  Reasons  for  Being  questions  (see  Schwerin, 
Michael,  &  Uriell  (2002)  for  a  description  of  Reasons  for  Being  and  for  actual  surveys). 
Results  can  be  presented  as  either  a  table  or  a  graph,  and  always  show  the  selected 
location  and  the  next  highest  level. 

In  the  situation  above,  the  user  is  a  claimant  user  from  the  Pacific  Fleet  (PACFLT). 
The  Location  box  lets  this  user  chose  among  the  bases  in  PACFLT,  including  Diego 
Garcia  and  Guam.  By  default,  all  locations  are  chosen,  but  the  user  can  chose  only  one 
base  or  a  select  number  of  bases  to  look  at. 

In  this  case,  the  user  is  interested  in  the  Navy  College  Program,  specifically  the  active 
duty  Navy  members  and  the  perceived  value.  No  filters  other  than  the  respondent’s  status 
are  being  applied. 
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This  table  shows  the  answers  chosen  by  respondents  for  the  selected  bases  (in  this 
case  all  bases  within  PACFLT)  and  for  the  next  higher  level  in  the  Chain  of  Command, 
the  entire  Navy.  This  allows  for  a  quick  comparison  to  determine  how  PACFLT  is  doing 
compared  to  the  entire  Navy.  In  this  example,  about  80  percent  of  PACFLT  is  getting  an 
expected  or  better  than  expected  value  for  their  dollar,  while  88  percent  of  the  entire 
Navy  feels  the  same  way. 
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This  shows  the  graphical  results  of  the  analysis.  With  the  graph,  it  is  easier  to  see  the 
comparison  between  the  levels.  While  more  of  the  Navy  (grey)  respondents  are  getting  a 
much  better  value  than  they  expected,  more  of  the  PACFLT  (black)  respondents 
indicated  they  don’t  know  or  it  is  not  applicable. 
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Program  Level  Analysis  Screen 
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Analyses  can  also  be  conducted  at  headquarters  for  an  entire  program.  This  shows  the 
analysis  screen  for  the  Child  Care  program.  Again,  the  user  can  select  specific  bases  or 
groups  of  bases  (default  is  the  entire  Navy)  as  well  as  the  respondent’s  status  and  three 
other  filter  questions. 
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Because  all  locations  in  the  Navy  were  selected,  there  is  no  higher  level  so  both 
numerical  columns  are  identical. 
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Reformatting  the  analyses  as  a  graph,  it  is  easy  to  see  that  the  two  groups  are  the 
same.  Had  a  specific  location  or  group  of  locations  been  selected,  the  black  bar  would 
change  to  indicate  data  for  the  selection,  while  the  grey  bar  would  remain  the  same  to 
indicate  program-wide  data. 


12 


•  Explain  requirements  at  each  level  of  leadership 


•  Have  a  mandate  for  change  from  leadership 

•  Collect  consistent  information 

•  Be  prepared  for  training  users 
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A  few  lessons  were  learned  in  creating  NEPTUNE.  Because  this  study  collected  data 
from  a  number  of  locations,  there  were  a  number  of  leaders  involved.  There  was  a 
perception  at  the  outset  that  because  two  locations  can  be  compared,  the  system  might  be 
used  as  a  “report  card.”  Briefings  at  each  of  the  locations  and  throughout  the  Chain  of 
Command  w'ere  conducted  to  explain  the  purpose  of  the  study  and  what  it  could  do  for 
them. 

The  lack  of  objective  data  was  discovered  early  in  the  study.  Because  there  was  no 
mandate  for  change  from  leadership,  the  researchers  received  the  objective  data  available 
and  had  to  find  a  way  to  make  it  work  instead  of  the  programs  rethinking  how  they  could 
better  collect  and  provide  data. 

One  of  the  needed  pieces  of  objective  data  was  number  of  users.  Programs  computed 
this  differently;  some  counted  how  many  came  through  the  door,  some  counted  how 
many  unique  users  they  had,  and  some  counted  how  many  drinks  they  sold.  There  was  no 
clear  definition  of  a  user. 

Training  in  interpreting  results  was  needed;  survey  data  typically  has  margins  of 
error,  so  a  5-percentage  point  difference  between  two  locations  may  not  indicate  that  one 
is  more  effective  than  the  other.  This  training  would  have  to  occur  at  all  access  levels  and 
for  all  users. 
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Navy  Customer  Satisfaction  DSS  -  New 
Possibilities 


NPRST 


•  New  standards  for  objective  and  subjective  data 

•  New  capabilities  utilizing  the  Navy-wide  Intranet 

•  New  possibilities  to  integrate  into  readiness  reporting 
system 


Nayj-  Personnel  Research.  Studies,  &  Technology 


Three  changes  have  occurred  since  the  beginning  of  this  project  that  could  create  new 
possibilities  for  a  customer  satisfaction  DSS.  First,  new  standards  are  being  developed  for 
programs  and  new  requirements  are  being  put  in  place  to  collect  objective  data  as  well  as 
subjective  data.  Second,  the  Navy  will  have  an  enterprise-wide  intranet  that  could 
enhance  DSS  capabilities.  Third,  the  Navy  is  creating  a  readiness  reporting  system  that 
will  be  widely  used,  and  a  customer  satisfaction  DSS  could  feed  results  into  this  system. 
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Reasons  for  Being 


1.  Promote  the  physical  and  psychological  wellbeing  of  members, 
maintaining  quality  of  life  at  a  level  to  attract  qualified  men  and 
women  to  the  Navy. 

2.  Promote  the  physical  and  psychological  wellbeing  of  members, 
maintaining  quality  of  life  at  a  level  to  retain  qualified  men  and 
women  to  the  Navy. 

3.  Provide  a  level  of  support  that  allows  members  to  concentrate  on 
their  mission. 

4.  Provide  a  level  of  support  that  allows  availability  for 
deployment. 

5.  Provide  educational  opportunities  that  lead  to  personal 
satisfaction,  maximization  of  individual  contributions,  and 
maintenance  of  the  expertise  required  for  the  Navy  of  the  future. 


.\a\y  Personnel  Research,  Studies,  &  Technology 


Reasons  for  Being 


6.  Demonstrate  concern  for  members  and  their  families  to  enhance 
morale  and  commitment  to  the  Navy. 

7.  Make  available  the  skills  and  tools  to  facilitate  personal 
relationships,  minimize  the  stresses  of  military  life,  and  help 
members  reduce  tensions  between  military  and  family  roles. 

8.  Help  to  ensure  the  health  and  safety  of  USN  personnel  and  their 
families. 

9.  Increase  personal  and  family  satisfaction  with  adaptation  to 
military  life  style. 


.Navy  Personnel  Research,  Studies,  &  Technology 
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B-0 


Appendix^ 


Chaplain  Religious  Enrichment  Delivery  Operation  (CREDO).  The  CREDO  program 
provides  an  assortment  of  ministries  to  sea  service  personnel  and  their  families  to  develop 
and  use  their  personal  and  spiritual  resources.  The  goal  of  CREDO  is  to  help  people  grow 
toward  increased  functional  ability,  spiritual  maturity,  and  acceptance  of  responsibility. 
CREDO  programs  include:  Personal  Growth  Retreat  (PGR);  Spirituality:  CREDO  II; 
Christian  Disciplines:  CREDO  III;  Reclaiming  the  Inner  Child;  Marriage  Enrichment 
Retreat  (MER);  and  others  (e.g.,  family  retreats,  teen  retreats,  men’s  retreats,  and 
women's  retreats).  CREDO  provides  personal  growth  training  in  nine  major  areas: 
Norfolk,  San  Diego,  Naples,  Okinawa,  Mayport,  Hawaii,  Camp  Pendleton,  Camp 
LeJeunc.  and  Bremerton.  Services  are  provided  to  active  duty,  reserve,  retired,  and  active 
duty  family  members. 

Child  Development  (MWR  program).  Base  Child  Development  provides  child 
development,  either  on  or  off  base,  for  children  aged  six  weeks  to  five  years.  Child 
Development  has  both  a  base  facility  and  secure  positions  with  in-home,  licensed 
providers.  Child  Development  also  offers  advice  and  guidance  for  families  exploring 
individual  child-care  providers.  Child  Development  programs  include  Child 
Development  Centers,  Family  Childcare,  Off-base  contract  centers,  and  Off-base  Family 
Childcare  centers. 

Clinical  Counseling  (FFSC  program).  The  Clinical  Counseling  program  offers  short¬ 
term,  individual,  marriage,  family,  and  group  counseling  to  address  situation  problems  in 
day-to-day  living,  depression/grief  after  a  loss,  troubled  relationships,  financial 
difficulties,  occupational  concerns,  and  family  issues  to  active  duty  Sailors  and  their 
dependents.  Counseling  may  take  the  form  of  education,  stress  management,  or 
workshops. 

Deployment  Support  (FFSC  program).  Deployment  Support  offers  assistance  to  active 
duty  Sailors  and  their  dependents  to  manage  the  challenges  of  deployment  (e.g., 
anticipate  and  understand  the  physical  and  emotional  demands  associated  with 
deployment).  Deployment  Support  consists  of  three  phases:  pre-deployment  support, 
mid-deployment  support,  and  retum/reunion  support. 

Food  and  Hospitality  (MWR  program).  Data  collection  sites  include  snack  bars  at  the 
golf  course  and  bowling  alley,  cafeteria  style  operations  at  the  Bachelor  Quarters, 
catering  operations  at  the  Officers’  club,  and  bar  lunch/dinners  operations  at  the 
Officers’,  Chief  Petty  Officer,  Enlisted,  and  Consolidated  clubs. 

Navy  College  Program  (NCP).  NCP  provides  consulting  services  for  military  members 
who  wish  to  gain  additional  education  skills  and  qualifications.  These  consulting  services 
include  tuition  assistance,  college/university  information  and  degrees  offered,  educational 


'  From  Schwerin,  M.  J.,  Michael,  P.  G.,  &  Uriell,  Z  A.  (2002).  Navy  Quality  of  Life  (QOL)  Program 
Contributions  to  Readiness  and  Retention.  Volume  1:  Design  and  Methodology  (NPRST-TN-02-5). 
Millington,  TN;  Navy  Personnel  Research,  Studies,  and  Technology. 
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goals  examined,  determining  what  past  training  may  be  applieable  to  a  degree  program 
and  assistanee  in  applying  for  entrance  to  a  particular  school.  While  it  does  not  provide 
educational  services  itself,  it  arranges  for  universities  to  open  campuses  on  or  near  base 
and  negotiates  degree  requirements.  NCP  serves  active  duty  with  the  following  programs: 
Academic  Skills,  Service  Member’s  Opportunity  Colleges  -  Navy  (SOCNAV,  2-  and  4- 
year  programs).  Defense  Activity  Non-Traditional  Education  Support  (DANTES), 
Program  Afloat  for  College  Education  (PACE),  High  School  Completion,  On-Base 
College  Program,  and  Tuition  Assistance  (TA). 

Nutrition  Education  and  Galiey  Food  Services.  The  Health  and  Physical  Readiness 
division  of  the  Navy  Personnel  Command  (PERS-60)  sponsors  a  nutrition  education 
program.  This  program  works  to  provide  dietary  education  to  Sailors  and  dependents 
while  assisting  Navy  galleys  in  menu  guidance.  The  local  base  commander  runs  the  Navy 
galley  food  services  system  with  assistance  from  the  Navy  Supply  Systems  Command. 

Personal  Financial  Management  (PFM FFSC  program).  The  PFM  program 
provides  personal  and  family  financial  education,  information  services,  and  assistance, 
including  but  not  limited  to  consumer  education,  advice  and  planning,  and 
savings/investment  counseling  to  active  duty  Sailors  and  their  dependents. 

Recreation  and  Fitness  (MWR  program).  Bases  provide  a  wide  variety  of  sports, 
recreation,  and  fitness  facilities  for  active  duty  military,  dependents,  retirees,  and 
government  civilians.  Programs  include;  Information,  Tickets  and  Tours  (ITT), 
Gym/Fitness,  Recreation  Center,  Single  Sailor  Program,  Intramural  Sports,  Library,  Park 
and  Picnic,  Outdoor  Recreation  Center,  Swimming  Pools,  Auto  Skills  Shop,  Bowling 
Center,  Riding  Stables,  Marinas,  and  Golf  Courses. 

Relocation  Assistance  Program  (RAP — FFSC  program).  RAP  offers  relocation 
information  to  active  duty  Sailors  and  dependents  as  well  as  government  civilians  for 
managing  the  military  lifestyle.  Services  include  destination  information,  intercultural 
relations  training,  settling-in  services,  help  finding  a  home,  and  school  information.  Other 
program  services  include:  Smooth  Move  (a  seminar  addressing  the  entire  relocation 
process).  Welcome  Aboard  seminars  (basie  training  for  Navy  or  Marine  Corps 
spouses/families  new  to  an  area),  and  Overseas  Transfer  Workshop  (topics  include 
overseas  screening,  dependent  entry  approval,  transportation,  schools,  household  goods, 
and  cultural  relations). 

Spouse  Employment  Assistance  Program  (SEAP — FFSC  program).  The  SEAP 
addresses  employment  needs  in  basic  workshops  (launching  a  job  search,  career 
planning,  resume  writing,  interview  techniques,  federal  employment  information,  and 
networking),  self-help  job  information  centers,  and  individual  employment  counseling. 
SEAP  serves  Navy  family  members,  retirees  and  their  spouses,  and  civilian  spouses  who 
are  relocating  overseas  to  Department  of  Defense  components. 

Transition  Assistance  Management  Program  (TAMP — ^FFSC  program).  TAMP  is 
designed  to  assist  service  members  with  career  planning  (i.e.,  weighini  civilian 
alternatives  against  their  military  careers)  and  their  transition  from  military  to  civilian 
life.  There  are  many  services  associated  with  TAMP,  however  the  most  widely  publicized 
is  the  Transition  Assistance  Program  (TAP),  a  seminar  designed  to  address  social, 
financial  and  professional  issues  associated  with  transition  out  of  the  military. 
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Youth  Programs  (MWR  program).  Base  youth  centers  provide  day  care  for  children 
aged  as  young  as  five  through  their  teens  with  a  variety  of  age  specific  recreation  services 
both  before  and  after  school  (all  day  during  the  summer).  Youth  Programs  includes  the 
Youth  Center,  Youth  Sports,  Teen  Activities,  and  Summer  Camps. 
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